The author describes creating a 340 M-parameter Llama-based model trained exclusively on English texts published before 1900. They built custom data pipelines, tokenization, base-training and fine-tuning scripts, handled deduplication and filtering of historical sources, and trained locally and on cloud GPUs for about $80. The result is a toy “Victorian” chatbot that can hallucinate and isn’t aligned for modern safety.

• Built a 340M-parameter LLaMA-style model trained exclusively on pre-1900 English texts, doing every step (data pipeline, tokenization, base-training, fine-tuning) from scratch
• Total cloud GPU cost was about $80, after prototyping a smaller 14M-parameter version locally
• Cleaning and deduplicating ~90GB of historic sources (Gutenberg, Internet Archive, British Library, Harvard) was the hardest and most time-consuming part
• The model has no hard 1900 knowledge cutoff or safety alignment, so it can hallucinate and let anachronistic facts (like Einstein) slip through